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PsJ Abstract 

The common method to estimate an unknown integer parameter vector in a hnear model is 
to solve an integer least squares (ILS) problem. A typical approach to solving an ILS problem is 
sphere decoding. To make a sphere decoding algorithm faster, the well-known LLL reduction is 
often used as preprocessing. The Babai point produced by the Babai nearest plan algorithm is a 
^! suboptimal solution of the ILS problem. First we prove that the success probability of the Babai 

point as a lower bound on the success probability of the ILS estimator is sharper than the lower 
bound provided by Hassibi and Boyd [• ] . Then we show rigorously that applying the LLL reduction 
^ algorithm will increase the success probability of the Babai point. Finally we show rigorously that 

f-H applying the LLL-reduction algorithm will reduce the computational complexity of sphere decoders 

which is measured approximately by the number of nodes in the search tree in the literature. 
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1 Introduction 



Consider the following linear model: 
Q y = Ax + v, (1) 

where y G M" is an observation vector, v € is a noise vector following the Gaussian distribution 
AA(0, a^I) with (T being known, A G ig a model matrix with full column rank, and ;e G is an 

unknown integer parameter vector. A common method to estimate x in (1) is to solve the following 
^ integer least squares (ILS) problem: 

. . min \\y — Axll'^, (2) 

^ whose solution x'^^ is the maximum-likelihood estimator of x. The ILS problem is also referred to as 
?H the closest point problem in the literature as it is equivalent to find a point in the lattice {Ax : x £ Z'^} 
which is closest to y. 

A typical approach to solving (2) is the discrete search approach, referred to as sphere decoding 
in communications, such as the Schnorr-Euchner algorithm [ ] or its variants, see for example [2, I]. 
To make the search faster, a lattice reduction is performed to transform the given problem to an 
equivalent problem. The widely used reduction is the LLL-reduction proposed by Lenstra, Lenstra 
and Lovasz in [10]. 

It has been shown that the ILS problem is NP-hard [18, 12]. Solving (2) may become time- 
prohibitive when A is ill conditioned, the noise is large, or the dimension of the problem is large 
[8]. So for some applications, an approximate solution, which can be produced quickly, is computed 
instead. One often used approximate solution is the Babai point, produced by Babai's nearest plane 
algorithm [ ]. This approximate solution is also the first integer point found by the Schnorr-Euchner 
algorithm. In communications, a method for finding this approximate solution is referred to as a 
successive interference cancelation decoder. 
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In order to verify whether an estimator is good enough for a practical use, one needs to find the 
probabihty of the estimator being equal to the true integer parameter vector, which is referred to as 
success probability [0]. The probability of wrong estimation is referred to as error probability, see, 
e.g., [7]. 

If the Babai point is used as an estimator of the integer parameter vector x in (1), certainly it is 
important to find its success probability, which can easily be computed. But even if one intends to 
compute the ILS estimator, it is still important to find the success probability of the Babai point. It is 
very difficult to compute the success probability of the ILS estimator, so lower and upper bounds have 
been considered to approximate it, see, e.g., [6, 20]. In [17] it was shown that the success probability 
of the ILS estimator is the largest among all "admissible" estimators, including the Babai point, which 
is referred to as a bootstrap estimator in [ i / ]. The success probability of the Babai point is often used 
as an approximation to the success probability of the ILS estimator. In general, the higher the success 
probability the Babai point, the lower the complexity of finding the ILS estimator by the discrete 
search approach. In practice, if the success probability of the Babai point is high, say close to 1, then 
one does not need to spend extra computational time to find the ILS estimator. 

Numerical experiments have shown that after the LLL reduction, the success probability of the 
Babai point increases ["]. But whether the LLL reduction can always improve the success probability 
of the Babai point is still unknown. In this paper, we will prove that the success probability of the 
Babai point will become higher after the LLL-reduction algorithm is used. It is well-known that 
the LLL reduction can make sphere decoders faster. But to our knowledge there is still no rigorous 
justification. We will show that the LLL reduction can always decrease the computational complexity 
of sphere decoders, an approximation to the number of nodes in the search tree given in the literature. 

The rest of the paper is organized as follows. In section 2, we introduce the LLL reduction to 
reduce the ILS probelm 2. In section 3, we introduce the Babai point and a formula to compute the 
success probability of the Babai point, and we show that the success probability of the Babai point 
is a sharper lower bound on the success probability of ILS estimator compared with the lower bound 
given in [G]. In section 4, we rigorously prove that the LLL-reduction algorithm improve the success 
probability of the Babai point. In section 5, we rigorously show that the LLL reduction algorithm 
reduce the computational complexity of sphere decoders. Finally we summarize this paper in section 
6. 

For X S M", we use [a;] to denote its nearest integer vector, i.e., each entry of x is rounded to its 
nearest integer (if there is a tie, the one with smaller magnitude is chosen). The success probabilities 
of the Babai point and the ILS estimator are denoted by Pb and -P/ls, respectively. 

2 LLL Reduction and transformation of the ILS Problem 

Assume that A in the linear model (1) has the QR factorization 

A=[Q^,Q^] 



where [Qi, Q2 ] S M™^™- is orthogonal and R € M"^" is upper triangular. Without lose of generality, 

n m—n 

we assume the diagonal entries of -R are positive throughout the paper. Define y = QiU- From (1), 
we have y = Rx + Q\v. Because v ~ AA(0, cr^/), it follows that y ~ M{Rx, a'^I). 
With the QR factorization of A, the ILS problem (2) can be transformed to 

mm \\y — -Ra;||2. (3) 

One can then apply a sphere decoder such as the Schnorr-Euchner search algorithm [' '] to find the 
solution of (3). 
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The efficiency of the search process depends on R. For efficiency, one typically uses the LLL 
reduction instead of the QR factorization. After the QR factorization of A, the LLL reduction [ i ] 
reduces the matrix R in (3) to R: 

cfRZ = R, (4) 

where Q G M"^" is orthogonal, Z G ^'"X" is a unimodular matrix (i.e., det(Z) = ±1), and R G M"^" 
is upper triangular with positive diagonal entries and satisfies the following conditions: 

\rik\<^rii, i = 1,2, . . . ,k - 1 (5) 

'^^fc-i.fe-i — + ^fcfc; k = 2,3,...,n, (6) 

where (5 is a constant with 1/4 < 5 < 1. The matrix R is said to be 5-LLL reduced or simply LLL 
reduced. Equations (5) and (6) are referred to as the size-reduced condition and the Lovasz condition, 
respectively. 

The original LLL algorithm given in [10] can be described in the matrix language. Two types of 
basic unimodular matrices are implicitly used to update R so that it satisfies the two conditions. One 
is the integer Gauss transformations (IGT) matrices and the other is permutation matrices, see below. 

To meet the first condition in (5), we can apply an IGT, which has the following form: 

Zik = I - Cejef. 

Applying [i < k) to R from the right gives 

R = RZ{k = R — ReiB^ . 

Thus R is the same as R, except that fjk = rj^ — C^jk for j = 1, 2, . . . , i. By setting C = [rik/ra] , we 
ensure \fik\ < fjj/2. 

To meet the second condition in (6) permutations are needed in the reduction process. Suppose 
that Sr^_^ > r'^_i k~^ "^"kk ^'-'^ some k. Then we interchange columns k — 1 and k of R. After the 
permutation the upper triangular structure of R is no longer maintained. But we can bring R back 
to an upper triangular matrix by using the Gram-Schmidt orthogonalization technique (see [10]) or 
by a Givens rotation: 

R = Gk-l,kRPk-l,k, (7) 
where Gk-i,k is an orthogonal matrix and Pk-i,k is a permutation matrix, and 

^k-l,k-l = ^fe-l,fc + '^fc.fcJ (8) 

-2 1-2 2 

^k-l,k "T ^k,k — ^k-l,k-l- 

Note that the above operation guarantees 5f'^_^ < f^_^ k^ ^kk since 5 < 1. The LLL reduction 
algorithm is described Algorithm 1, where the final reduced upper triangular matrix is still denoted 
hyR. 

After the LLL reduction (4), the ILS problem (3) is then transformed to: 

min (9) 

where y = Q y and z = Z x. 

The LLL-reduction is a powerful preprocessing tool that allows to reduce the complexity of search 
process for finding the ILS solution, see, e.g., [6, 2]. 
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Algorithm 1 LLL reduction 



1: compute the QR factorization: A = Q 



R 




set Z = In, k = 2; 
while k < n do 

apply IGT Zk-i± to reduce r^-i^k- R = RZk-\,k\ 

update Z: Z = ZZk-i.y, 

if '^^'Li,fc-i >^Li,fc + ^'Ifcthen 

permute and triangularize R: R = Gk-i,kRPk-i,k', 
update Z: Z = ZP^^i^^] 
k = k — 1, when k > 2; 
else 

for i = k — 2, . . . ,1 do 

apply IGT Zik to reduce rj^: R = RZik] 
update Z: Z = ZZiy, 
end for 
k = k + l] 
end if 
end while 



3 Success Probability of the Babai point and a lower bound 

The first integer point found by the Schnorr-Euchner search algorithm [14] for solving (3) is the Babai 
point a;^ [ ], which is defined as follows: 

C-n — yn/fnrii — [Cji] , 

Ci = {yk- rijxf)/rii, xf=[ci], 
j=i+i 

for i = n,n — 1,...,1. Note that the entries of are determined from the last to the first. 

Before we give a formula of the success probability of the Babai point, we would like to introduce 
Teunissen's result given in [16]. It is easy to verify that the ILS problem in (2) is equivalent to 

m.m(x-x'''^^fW-\x-x''''^), (11) 

where a;^^^ = {A^ A)^^A^y is the real LS estimator of the integer parameter vector x in (1) and 
W = a^{A'^ A)~^ is its error covariance matrix. In the GPS literature, the form (11) is often used 
instead of (2). Let W have the LDL""" factorization: W = LDL^, where L is unit lower triangular 
and D is diagonal. Then based on this factorization, the Babai point, which is referred to as the 
Bootstrapping estimator by Teunissen can be generated. Unlike x^, the entries of this Babai point 
are determined from the first to the last due to the form of the factorization of W. The formula of 
the success probability of this Babai point given in [ ] is Y\i=i 1^ J^/C^^^') exp(— ^t^)dt, where da 
is the diagonal entries of D. 

Given A and y in the linear model (1), it is more computationally efficient and reliable to compute 
the QR factorization of A than to compute the LDL"*" factorization of (A'^A)~^ in finding the ILS 
estimator of ;» in (1). In the following we give a formula for the success probability of this Babai point 
x^ . For the sake of readability, we give a proof, which is easier to follow than the proof given in [ 1 0] 
for the formula mentioned in the previous paragraph. 
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Theorem 1 Suppose y ^ N'{Rx,a'^I) in the ILS problem (3). Let Pg denote the success probability 
of the Babai point given in (10), i.e., Pb = P{x^ = x). Then 

Pb = JlHr^^), HO = \l ' jj^ exp{--t'')dt. (12) 
Proof. By the chain rule of conditional probabilities: 

n 

Ps = P{x^ =x)= P(f|(LQl = Xi)) 

i=l 

n-1 

= P{[Cn\ = Xn) P{[Ci \ = Xi\[ci+l \ = Xj+i, • • • , [Cn\ = X„). (13) 

Since y ~ Af{Rx, a"^!), 



Thus, 



Vn ^ NirnnXn.cr'^), tji ^ M{riiXi+ ^ rikXk,cr^), i = n - l,n - 2, . . . ,1. 

k=i+l 



Cn = Vn/rnn ^ J^iXn,(T /r^J, 
n 

Ci = {Vi- ^ 'rij[cj\)/rii ~ Mixi.a"^ /rfi), if [cj+i] = Xj+i, • • • , [cn\ = Xr. 
j=i+i 



Then it follows that 



PilCn] = Xn) = / ^ exp(- ^ )(it 



1 .2 



1 /"a . r 



r„„ " 2 ^ '"n.n ' 

/ 2(T 1 „ 

/ exp(--i2)dt 

■70 ^ 



Similarly, we can obtain 

P([q] = fj|[ci+i] = Xi+i, • • • , [c„] = x„) = 4>{rii). 

Then from (13) we can conclude that (12) holds. □ 

The success probability -P/^s of the ILS estimator depends on its Voronoi cell [' ] and it is difficult 

to rigorously compute it because the shape of Voronoi cell is complicated. In [ ] a lower bound 

S 

F{^^,n) is proposed to approximate it, where dmin is the length of the shortest lattice vector, i.e., 
dmin = iiiiiioT^aieZ" ||-R^c||2) and F is the cumulative distribution function of chi-square distribution. 
However, no polynomial-time algorithm has been found to compute drain- To overcome this problem, 
[6] proposed a more practical lower bound i^( ^§ , n) , where rmin = 

miuj ra . Note that Pb is a lower 
bound on P,i^s (see [1 (]). The following result shows Pb is sharper than F{-^,n). 



Theorem 2 



r2. 

mi 

V4ct2 
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Proof. Let u ~ A/'(0,/„). Thus ui,U2, ■ ■ ■ ,Un are i.i.d. and YA=i'^i follows the chi-squared 

2 2 

distribution with degree n. Let events E = {Y^^=i i^f < 4p^} ^^'^ ~ — i^l foi' * = l, 2, . . . , n. 
< r^i, ^ C niLi^i- Thus, 



2 n 71 



i=l 1=1 



1 /■-'^ -j^ 



1=1 2<T 



n 



i=l 



□ 

In the following, we give an extreme example to show that F^^p^,n) can be much smaller than 

Pb. 



Example 1 Let R 



0.001 
10 



and a = 0.5. By simple calculations, we obtain Fl -f^,n] /Pg 



Acr^ 

j^QQ- Although this is a contrived example, where the signal-to-noise ratio is very small, it shows that 
Pb can be much sharper than F^^p^,n^ as a lower bound on Pjls- 

4 Enhancement of by the LLL reduction 

In this section we rigorously prove that the LLL reduction algorithm can enhance the success proba- 
bility Pg of the Babai point. 

Suppose we have the QRZ factorization (4), where Q is orthogonal and Z is unimodular (we do 
not assume that R is LLL reduced unless we state otherwise). Then with y = Q y and z = Z~ x 
the ILS problem (3) can be transformed to (9). For (9) we can also define its corresponding Babai 
point z^ . This Babai point can be used as an estimator of z = Z^^x, or equivalently Zz^ can be 
used an estimator of x. In (3) y ~ M{Rx,a'^I). It is easy to verify that in (9) y ~ M{R.z,a'^I). In 
the following we look at how the success probability of the Babai point changes after some specific 
transformation of R. 

The following result shows that if the Lovasz condition (6) is not satisfied, after a column permu- 
tation and triangularization, the success probability of the Babai point increases. 

Lemma 1 Suppose that 6r'^_^ > t^_-^ ^ + r'^j^ for some k for the R matrix in the ILS problem 
(3). After the permutation of columns k — 1 and k and triangularization, R becomes R, i.e., R = 
Gk~i,kRPk~i,k (see (7)). Then with y = Gjl_ij^y and z = P^^^^^x, (3) can be transformed to (9). 
Denote z = P^^i f,x. Then the Babai z^ has a success probability greater than or equal to the Babai 
point x^ , i.e., 

P{x'' = x)<P{z^ = z), (14) 
where the equality holds if and only if r^^i^j. = 0. 

Proof. By Theorem 1, what we need to show is the following inequality: 

n n 

J{(t>{ru) <\{(t>{ni). (15) 

i=l i=l 
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Since ra = ru for i ^ k — 1, k, we only need to show 
which is equivalent to 



/ exp(--t2)(it / exp(--f2)dt 

f 2cr 1 „ /■ 2ct 1 „ 

< / exp(--r)dt / exp(--r)dt 

Jo 2 Jo 2 



Since Gk-i.k is orthogonal and P^-i^k is a permutation matrix, the absolute value of the determi- 
nant of the submatrix Rk-i-k.k-i-.k is unchanged, i.e., we have 

rk-i,k-irkk = rk-i,k-irkk- (17) 

Let 

rk-i,k-irkk rk^ik^ifkk .,„^ 

a = ' = , 18 



/(C) = In/ eM-lt^)dt + ln r expi-lf)dt. (19) 
JO ^ JO ^ 

Then (16) is equivalent to 



^^ max{rfc-i,fc-i,rfcfc} ^ ^ ^ ^ max{rk-i,k-i,rkk} 'j ^^g) 

Obviously, if r^-i^k = 0, then the equality of (20) holds since in this case ™'^^i^fc-^i^fc-i''"fcfc} — 
max{rfc^^y„i,rfcfc} ^ ^^^^ need to show if rk-i^k / 0, then the strictly inequality of (20) holds. In 

the following, we assume r^-i^k 7^ 0. 

From 5r'j,_^ ^ -^ > r'^_i ^ + and (8) we can conclude that 

rkk,rk-i,k-i,rkk < rk-i,k-i- 

Then, with (18) it follows that 

max{rfe_i fc_i,rfcfc} rk-i,k-i max{ffc_i yt_i, ffc^} , ^ 
2a 2a 2a ~ ^ 

Thus, to show the strictly inequality of (20) holds, it suffices to show that when (" > -y/a, /((") is a 
strictly monotonically decreasing function or equivalently /'(C) < 0. 
Prom (19), 

exp(-lC^) ^exp(-¥) 



/o exp(-lt2)dt J^Cgxp(_1^2)^i 

1(.(C)-.Q)). 



where g{C) = } i^L, - ^ > ^ > Thus, in order to show /'(C) < for C > \/a, 



Cexp(-IC^) 
/o«exp(-|t2)dt- 

we need only to show that g{() is a strictly monotonically decreasing function or equivalently ^'(C) < 
when C > 0. 
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Simple calculations give 



exp( 



(1 - e) j^^ eM-\t')dt - C exp(-^C') 



If 1 - < and C > 0, then obviously ^'(C) < 0. If 1 - > and C > 0, since ey.v{-\t^) < 1, 

(1 - C') j\M-\t')dt < C(l - C') < Cexp(-^C'), 

where the second inequality can easily be verified. Thus again g'{C) < when C > 0; completing the 
proof. □ 

Now we make some remarks. The above proof shows that /(C) for ^ > ^/a reaches its minimum 
when = y/a. Thus if fk-i,k-i = ^kk^ Pb will increase most. 

In Lemma 1 there is no requirement that Vk-i^k should be the size-reduced. The question we would 
like to ask here is do size reductions in the LLL reduction algorithm affect Pg? From (12) we observe 
that Pb only depends on the diagonal entries of R. Thus size reductions alone will not change Pb- 
However, if a size reduction can bring changes to the diagonal entries of R after a permutation, then 
it will affect Pb- Therefore, all the size reductions on the off diagonal entries above the superdiagonal 
have no any effect on Pg. But the size reductions on the superdiagonal entries may affect Pb- There 
are a few different situations. 

Suppose that the Lovasz condition (6) holds for a specific k. If (6) does not hold any more after 
the size reduction on rfc_i fc, then columns k — 1 and k of R are permuted by the LLL reduction 
algorithm and according to Lemma 1 Pb increases. If (6) still holds after the size reduction on rk~i^k, 
then this size reduction does not affect Pb- 

Suppose the Lovasz condition (6) does not hold for a specific k. Then by Lemma I Pb increases 
after a permutation and triangularization. If fc| > r,fc_i ,fc_i/2, in the next lemma, we show that 
the size reduction on rk~i.k performed before the permutation can increase Pb further. 

Lemma 2 Suppose that in the ILS problem (3) R satisfies > ^'f.-ifc + ^tk '^"'^ k/fc-i.fel > 

rfc_i^fe_i/2 for some k. Let R, y, z and z be defined as in Lemma 1. Suppose a size reduction on rk~i,k 
is performed first and then after the permutation of columns k — 1 and k and triangularization, R 
becomes R, i.e., R = Gk^i^kR^k^i^^Pk^i^k- Let if = Gj^_i f,y and w = P^\ ^^Z],^^ f^x, then (3) is 
transformed to min^jg^n ||y — Rw\\2. Denote w = P^\ f^Z'^^-^ ^x. Then the Babai corresponding 
to the new transformed ILS problem has a success probability greater than or equal to the Babai point 
z^ , i.e., 

P{z'' = z) < Piw" = w), (21) 
where the equality holds if and only if 

\rk-i,k-irk-i,k\ = rLi.fe + rlk- (22) 
Proof. Obviously it suffices to show that 

4'{rk-i,k~i)(t>{rkk) < <P{rk-i,k-i)4>{rkk), 

which, by the proof of Lemma 1, is equivalent to 

/(max{ffc_i,A.._i,ffcfc}) < /(max{rfc_i,fc_i,ffcA,.}), 

where / is defined in (19). Since /(C) has been showed to be strictly monotonically decreasing when 
C > \/a. what we need to show is 

max{ffc_i^fc-i, rkk} > Taaii{rk~i,k-i,rkk}, (23) 
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and the equality holds if and only if (22) holds. 
Since |rfc_i,fc| > rfc„i,fe„i/2, 



rk-i,k-i - \jT'k-i,k + '^kk > \J'^k-i,k-i/^ + ^/cfc' 

rk~i,k-irkk rk-i^k-irkk 
rkk = I = < 



k~l,k~^^kk v ^I-i,fe-i/^ + ^fcfe 



But ./rti ,_i/4 + > , "'^■''-7" 2 ' ^^^^ 

max{fA:_l,A;-l,rfcfc} = ffc-l.fc-l. 

Suppose that after the size reduction, r^.i ^ becomes fk-i k- Note that 



ffc_i,fc_i = Jrl_,k + '"L < \rk-i,k + rlk = fk-i,k-i- 



Thus, it follows from (23) what we need to prove is fkk < i'k-i,k-i or equivalently 

hk < ^rl_^,^+rl 



kk^ 



and the equality holds if and only if (22) holds. 

By the conditions given in the lemma, |rfc_i^fc| < rk-i,k-i < '^\fk-i,k\- Thus 



If rk-i,k > 0, then 



fk-i,k = rk-i,k - \ rk-i,k/rk-i,k-i\rk-i,k-i 
= rk-i,k - sigu{rk-i,k)rk~i,k-i- 



rk-i,k-irkk rk-i,k-irkk 

rkk 



rk-i,k-i /f 2 I J.2 

\j'k~l,k^'kk 

rk-l^-lTkk 



\J{rk-i,k -rk-i,k-iY + 



2 

kk 



Thus it suffices to show 



fk-i^k-irkk <- I \ -i 



{rk-i,k - rk-i,k-i)'^ + 
Simple algebraic manipulations shows that the above inequality is equivalent to 

{rk-i,k~irk~i,k - rl_i^k - ^Ikf > 0, 
which certainly holds. And obviously, the equality in (24) holds, if and only if 

rk~i,k~irk-i,k = r-Li,fe + rlk- 
If Vk-i^k < 0, we can similarly prove (24) holds and the equality holds if and only if 

-rk-i,k-irk-i,k = rLi.fc + rlk, 

completing the proof. □ 

From Lemmas 1 and 2 we immediately obtain the following results. 



Theorem 3 Suppose that the ILS problem (3) is transformed to the ILS problem (9), where R is 
obtained by Algorithm 1. Then 

P{x^ = x) < P{z^ = z), 

where the equality holds if and only if no column permutation occurs during the LLL reduction process 
or whenever two consecutive, say k — 1 and k, columns are permutated, r^-i^k = 0. Any size reductions 
on the superdiagonal entries of R which is immediately followed by a column permutation during the 
LLL reduction process will enhance the success probability of the Babai point. All other size reductions 
have no any effect on the success probability of the Babai point. 

Now we make some remarks. Note that the LLL reduction is not unique. Two different LLL 
reduction algorithms may produce different R. In Algorithm 1, when the Lovasz condition for two 
consecutive columns is not satisfied, then a column permutation takes places to ensure the Lovasz 
condition to be satisfied. If an algorithm which computes the LLL reduction does not do permutations 
as Algorithm 1 does, e.g., the algorithm permutes two columns which are not consecutive or permutes 
two consecutive columns but the corresponding Lovasz condition is not satisfied after the permutation, 
then we cannot guarantee this specific LLL reduction will increase Pg. 

It is interesting to note that [II] showed that all the size reductions on the off diagonal entries 
above the superdiagonal of R have no any effect on the residual norm of the Babai point. Here we see 
that those side reductions are not useful from another perspective. 

By Lemma 1 , a column permutation of R can increase Pb ■ A large 5 is likely to increase the chances 



of column permutations and so increase Pb- Let A have the QRZ factorization QJ AZ 



Ri 





where 



Ri and R2 are (^i-LLL reduced and 52-LLL reduced, respectively, and 5i < 82. Note that R2 must 
be (5i-LLL reduced, but -Ri may not be (52-LLL reduced. If R2 can be obtained by applying the LLL 
reduction algorithm (with 5 = 82) io Ri, then by Theorem 3 the success probability of the Babai 
point corresponding to R2 should be at least as large as the success probability of the Babai point 
corresponding to Ri. However, the conclusion may not hold in general if Ri and R2 were obtained 
by applying the LLL reduction algorithm with 8 = 81 and 8 = 82 respectively to A. The reason is the 
LLL reduction is not unique. We use an example to illustrate this. 



Example 2 Let 



Applying Algorithm 1 to A with 8 
82-LLL reduced R2, respectively: 



0.9675 




= = 



0.4328 
0.5879 





I and 8 



0.0935 0.9477 

0.6792 0.4456 

0.4295 0.0549 

0.0853 



25 
36' 



we obtain 81-LLL reduced Ri and 



R^ 



R2 



0.4852 





0678 


-0.0232 


-0.1236 





-0 


3485 


-0.0578 


0.1235 










-0.3413 


0.0990 













-0.3612 


0.5549 


-0 


2591 


0.1280 


-0.0001 








3845 


-0.1278 


-0.0855 










0.2960 


-0.0996 













0.3299 
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Note that Ri is not 82 LLL reduced. Applying Algorithm 1 to Ri with 5 = 82, we obtain 52-LLL 
reduced R2: 

r0.3550 0.0523 -0.1448 0.0926" 

0.3429 -0.0889 -0.0470 

0.3767 -0.1346 

-0.4544 



R7 



Let a = 0.1, then we have 

Pb{Ri) = 0.7665, Pb{R2) = 0.7295, Pb{R2) = 0.7756. 
Obviously, Pb{R2) < Pb{Ri) < Pb{R2)- 

5 Reduction of the search complexity by the LLL reduction 

In this section, we rigorously show that applying the LLL reduction algorithm can reduce the compu- 
tational complexity of sphere decoders, which is measured approximately by the number of nodes in 
the search tree. 

The complexity results of sphere decoders given in the literature are often about the complexity 
of enumerating all integer points in the search region: 

\\y-Rx\\2<l3, (25) 

where /3 is a constant called the search radius. A typical measure of the complexity is the number of 
nodes enumerated by sphere decoders, which we denote by C,. 
For i = n, 71 — 1, . . . , 1, define Ei as follows 

Ei = \{Xi.,n G : \\y^,^ - Ri..n,r.nXi:nh < (26) 

As given in [ ], Ei can be estimated as follows: 

I det(J?j;j2^j;j^) I Irjjrj-i-i^j+i • • • rnn\ 

where Vn-i+i denotes the volume of an (n — i + l)-dimensional unit Euclidean ball. This estimation 
would become the expected value to Ei if is uniformly distributed over a Voroni cell of the lattice 
generated by Ri:n,i:n- Then we have (see, e.g., [ , Sec 3.2] and [ >]). 

" " T/ an—i+1 

C = ^E,^C{R)^E^^''^ ■ (28) 

1=1 i=\ 

In practice, when a search decoder such as the Schnorr-Euchner algorithm is used in the search process, 
after an integer point is found, (5 will be updated to shrink the search region. But C C here do not 
take this into account for the sake of simplicity. 

The following result shows that if the Lovasz condition (6) is not satisfied, after a column permu- 
tation and triangularization, the complexity C{R) decreases. 

Lemma 3 Suppose that 5r1_^ > r1_-^ ^ + r^^ for some k for the R matrix in the ILS prob- 
lem (3). After the permutation of columns k — 1 and k and triangularization, R becomes R, i.e., 
R = Gk-i,kRPk-i,k (see (7)). Then the complexity C,{R) of the search process decreases after the 
transformation, i.e., 

C{R) > C{R). (29) 
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Proof. Since ru = ra for i / /c - 1, k, rk-i^k-iTkk = rk-i,k-irkk, and r^k > rkk, we have 

" T/ on—i+1 " T/ an— i+1 

an) - m = y ^ T — 



\r an—k+l T/ on— fe+1 

Vn-k+1 P Vn-k+1 P 



fkkT^k+l,k+l ■ ■ ■ fnn fkkfk+l,k+l " 
1 1 ^ Vn-k+1 ^ Q 



.''fcfc ''fcfc / ^/c+l./c+l ■ ■ ■ Tnn 
□ 

Suppose the Lovasz condition (6) does not hold for a specific k and furthermore |rfc_ifc| > 
rfc_i^fc_i/2. The next lemma, which is analogous to Lemma 2, shows that the size reduction on 
r/c_i performed before the permutation can decrease the complexity C,{R) further. 

Lemma 4 Suppose that in the ILS problem (3) R satisfies > r'^_ij^ ~^ '''tk '^"'^ kfe-i.fel > 

rfc_i^fc_i/2 for some k. Let R he defined as in Lemma 3. Suppose a size reduction on r^-i^k is performed 
first and then after the permutation of columns k — 1 and k and triangularization, R becomes R, i.e., 
R = Gk-i,kRZk-i,kPk-i,k- Then 

CiR) > CiR). (30) 
Proof. By the same argument given in the proof of Lemma 3, we have 

C(i^) - C{R) ' ^ ^ ^ 



rkk rkk J rk+i^k+i ■ ■ ■ rnn 

To show (30) we need only to prove fkk < rkk- Since fk-i,k-irkk = rk-i,k~irkk and fk-i^k-i < ^fe-i,fe-i 
(see (??) in the proof of Lemma 2), we have fkk < rkk, completing the proof. □ 
From Lemmas 3 and 4 we immediately obtain the following result. 

Theorem 4 Suppose that the ILS problem (3) is transformed to the ILS problem (9), where R is 
obtained by Algorithm 1. Then 

C{R) > CiR), 

where the equality holds if and only if no column permutation occurs during the LLL reduction process. 
Any size reductions on the superdiagonal entries of R which is immediately followed by a column 
permutation during the LLL reduction process will reduce the complexity (. All other size reductions 
have no any effect on 

The about result on the effect of the size reductions is consistent with a result given in [1 9], which 
shows that all the size reductions on the off-diagonal entries above the superdiagonal of R and the 
size reductions on the superdiagonal entries of R which are not followed by column permutations have 
no any effect on the search speed of the Schnorr-Euchner algorithm for finding the ILS solution. 



6 Summary and future work 

We have shown that the success probability Pb of the Babai point will increase and the complexity ( of 
sphere decoders will decrease if the LLL reduction algorithm given in Algorithm 1 is applied for lattice 
reduction. In addition, we have shown that Pb is a better lower bound on the success probability of 
ILS estimator than the lower bound given in [ ]. 

The implementation of LLL reduction is not unique. The KZ reduction [ ] is also an LLL reduction. 
But the KZ conditions are stronger than the LLL conditions. Whether some implementations of the 
KZ reduction can always increase Pg and decrease ^ will be studied in the future. 
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